subtitle: even the best-laid plans…
Last year (spring 2015) we collected a really nice set of data of students collaborating in groups of three. The data collection process wasn’t entirely smooth or perfect, but it generally went off without any major technical or logistical problems. We ended up with a really nice dataset of almost 150 students with high quality audio data (four channels per group), video recordings (one per group), and computer log files (ideally one per group, practically more than one). [NB: The annotated audio from this first phase of data collection will be made available soon to other researchers. You can read the paper about the data set (presented at Interspeech 2016) here.]
In the spring of 2016 we set off to do our second phase of data collection, in classrooms during a regular class period. So unlike the first phase where we had just two groups at a time with kids who had volunteered and were excited to try out some math problems (a.k.a. the best kids), we had up to 10 groups at once with varying levels of excitedness and/or willingness to follow directions. We mostly wanted to test out how well the audio capture worked with all of the background noise in a typical classroom environment and see if our speech models still held up.
Let’s get practical
One of the things that I get asked the most is which microphones we used in our data collection. We initially used one type of microphone for our pilot work which seemed fine at the time. However, by the time we got funding and needed to buy microphones, a newer one was available and it was waaaaaayyyy better. It’s a close-talking head-worn directional microphone. Here’s a link to it on Amazon.
It’s $80, which is inexpensive from some points of view and very expensive from others. Obviously, this microphone is not going to be a solution for scaling up this solution to lots and lots of classrooms. But it is entirely feasible for our data collection needs, where we only need to buy one classroom set and cart it around with us. We used the wired version of the microphones, but they also make a wireless version, which sounds easier in a classroom setting (no cord!) but it’s actually much more difficult to use when you have a whole classroom of kids using them at the same time. (Basically, it’s because they are wireless and need frequency space to transmit their data to a recording device nearby and if you have too many wireless devices trying to transmit you won’t have enough available frequency bandwidth available.) So, the wireless version was a non-starter at this time in available technology and we went with the logistically easier wired versions.
The computer-based collaborative math activities were delivered on iPads, using a custom built app. This app had some tech issues here and there, but overall worked pretty well. It logged the different choices that kids made while solving the problems and gave some minimal feedback as well. Each student in the group of three had a controller (a hacked Wii nunchuck) that was plugged into a base station for the iPad.
Wifi troubles
In order to store the logged data, our app needed an internet connection. We thought about ways to do it without one, but it was going to be even more complicated and we would be at risk of losing a bunch of data most likely. So we needed to get on the wifi at each school.
If you’ve done any research at a school anytime in the last decade or so, you know that the internet situation is, let’s say, not good. This is not news. And it hasn’t gotten appreciably better in many of the schools I’ve been to. Sometimes the internet is so locked down that it’s basically unusable. If, for instance, we wanted students to have access to our app, then the server that the app communicated with (to log data) would probably need to be whitelisted ahead of time by the school’s IT person. (This is usually a task that takes way longer than it should.) Other times, it requires a log in to a guest wifi that is typically overloaded with students trying to use Snapchat. (Once I also did some work at a school where the teacher had to personally log in to each device we were using because there was no easy way to get guest access and she didn’t trust anyone with her login info.)
Unreliable wifi means that the app was going to have some lags and delays in it, which is annoying for the students (and us) but also can cause other problems. If the lag was really bad, then we typically had to restart the iPad or at least the app. This usually helped. But it meant that there was now a second (or third) log file for that group (because a new app session had started). So that means that we then later had to sync up mulitple log files for lots of groups. Luckily the log files had a really nice timestamp in them so this wasn’t too difficult to script, but it was another thing that had to be noted during data collection and dealt with during data management/cleaning later.
Trusting batteries (and not)
In our very first pilot data collection many years ago, we used our audio receivers on battery mode because a) it was more convenient to not have to worry about where we were placing the receiver and if it was close enough to an outlet and b) the receiver had a battery mode and that seemed great. At some point during the 30 minute session, the receiver turned off. The batteries failed (even though they were new) and the receiver turned off without any warning. We were going around to check the system every 5-10 minutes or so, so we don’t know exactly how long it was off for (and not recording). This was unacceptable, so we decided for all future data collection we would use power cords and plug them in (and use a battery backup) even though that would be much more annoying from a practical standpoint. We just couldn’t risk losing any data due to something silly like that.
This worked relatively well! We only had one issue when a kid knocked into a cord and unplugged it. Luckily, with the battery backup, we only lost a couple minutes of data.
The GoPros had decent batteries that we trusted, so we went with battery-only for those to minimize the cord situation. But there were still a lot of cords.
So many cords
Like, a lot of cords. Each desktop set up required about 10 cords. We labeled each cord and had some print-outs to help the researchers when setting up each station. (Also, each person going out for data collection had to attend a training session to learn about what the activity looked like, how they could or could not intervene, how to troubleshoot tech problems, and most importantly how to set up the stations.)
Here’s a picture of what it looked like:
Data synchronization and planning
Maybe this section should be higher up in the post. It’s probably the most important thing to think about, and to think about as early as possible in your planning process. But it’s also less “practical” than some of these other things and is more subjective based on what kinds of multi-modal data you are collecting and how you are collecting it. But, it’s really important.
We learned a lot about data synchronization after our first phase of data collection. We had done quite a bit of planning to try and think of synchronization issues, but we hadn’t thought of everything. Between our first phase of data collection last year and our second phase this year, we made a few changes to our software and some additional changes to procedures and hardware.
…And then that didn’t work in practice. We tried it twice, with varying degrees of success. The cable monstrosity worked fine for maybe about 40% of groups. Another 30% of groups had lots of lag but were still able to make it work for at least a while, and the other 30% just couldn’t do anything. So that was fun. After another hour of fiddling we decided that the cable monstrosity was just not worth the headache, especially since it only worked like 40% of the time, and we would find a more analog solution for our synchronization problem. Now, perhaps with a better app or more time or higher quality cables this solution would have worked, but maybe not. We decided for the more manual approach (writing down what time exactly you started the audio recording) and devised some backup procedures for when the inevitable happens and someone forgets to do it (it happens to the best of us sometimes). We ended up hand-aligning the video and audio data streams like we had in the first round of data collection. I would still like an automatic solution for this problem, but at least we can relatively easily do it by hand for now. (Obviously the hand-aligning approach does not scale – but also this alignment – audio and video – is only necessary in order to do the qualitative human coding of the files which wouldn’t be scaled anyway.)
Controlled chaos
On its best days, a middle school classroom can be a bit chaotic. This is not necessarily a bad thing! It can mean that kids are discussing things with each other and figuring stuff out. (It can also mean that kids are throwing wadded up pieces of paper at one another or checking their smuggled-in cell phones when the teacher turns her back.) Bringing in a bunch of iPads, microphones, and GoPros to try out a collaborative math activity is just adding to the chaos that is already bubbling underneath the surface. With this kind of situation is it really important to have a good team of researchers that is familiar and comfortable with working in middle school classrooms and equally important to work with a teacher that is comfortable with this kind of controlled chaos. Good teachers usually are, but it’s always important to communicate beforehand (and during the activity) with the teacher so they know what is in store and how they can help. Remember, this is their classroom and they are the authority figure and you want to work with them.
In the future
One of the things I have been thinking about a lot while doing this work is how this will look (and work) in the classroom of the future. Improving learning and making something that can actually be used in real classrooms by real teachers and students is the goal, without any intervention by researchers. Maybe that future is 3 years away, maybe it is more like 8-10 years away. I’m not sure.
One issue that always concerns me about using speech data in real classrooms is the idea of some black box listening to students all the time and having them be worried about what it is tracking and how it is using that data. This can affect the general culture of the classroom and I don’t want that kind of anxiety present unnecessarily. I want students to know when and how we are collecting data about their learning. Initially we thought that we would transition away from the headset microphones because they are quite obvious and are a thing that kids need to remember to put on (and not fuss with) while working on the collaborative activities. But after talking over it with a teacher this past summer, we decided that we actually liked the idea of it being an obvious marker of data collection. That the microphne signaled to the student that we would be recording/monitoring what they were saying. This should hopefully avoid most of the issues around the spying/Big Brother aspect of it. And, as a bonus, we can also signal the beginning of collaboration and tell the students to “Put on your headsets, it’s time to collaborate!”
Cynthia D'Angelo 